Papers with pretraining method
OPAL: Ontology-Aware Pretrained Language Model for End-to-End Task-Oriented Dialogue (2023.tacl-1)
Copied to clipboard
| Challenge: | Existing task-oriented dialogue systems lack ontology-aware pretraining methods for task-orientated dialogue. |
| Approach: | They propose an ontology-aware pretrained language model (OPAL) for end-to-end task-oriented dialogue (TOD) . they propose to pretrain on large-scale contextual text data to bridge the gap between the pretraining method and downstream tasks. |
| Outcome: | The proposed model achieves an exciting boost and obtains competitive performance even without any TOD data on CamRest676 and MultiWOZ benchmarks. |
A Little Pretraining Goes a Long Way: A Case Study on Dependency Parsing Task for Low-resource Morphologically Rich Languages (2021.eacl-srw)
Copied to clipboard
| Challenge: | Neural dependency parsing has been a success for many domains and languages, but the bottleneck of massive labelled data limits its effectiveness for low resource languages. |
| Approach: | They propose to use morphological knowledge to improve dependency parsing for morphology rich languages in a low-resource setting to perform experiments. |
| Outcome: | The proposed method achieves an average gain of 2 points (UAS) and 3.6 points (LAS) on 10 MRLs in low-resource settings. |
Multi-CLS BERT: An Efficient Alternative to Traditional Ensembling (2023.acl-long)
Copied to clipboard
| Challenge: | ensembling BERT models often improves accuracy but at the cost of significantly more computation and memory footprint. |
| Approach: | They propose a new ensembling method for CLS-based prediction tasks that is almost as efficient as a single BERT model. |
| Outcome: | The proposed method outperforms existing BERT models on GLUE and SuperGLUE with 100 training samples. |
LICHEE: Improving Language Model Pre-training with Multi-grained Tokenization (2021.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained language models are trained based on single-grained tokenization, making it hard to learn the precise meaning of coarse-grain words and phrases. |
| Approach: | They propose a language model pretraining method that incorporates multi-grained information of input text into pre-trained language models. |
| Outcome: | The proposed method improves performance on CLUE and SuperGLUE in Chinese and English with little extra inference cost. |
ReactXT: Understanding Molecular “Reaction-ship” via Reaction-Contextualized Molecule-Text Pretraining (2024.findings-acl)
Copied to clipboard
Zhiyuan Liu, Yaorui Shi, An Zhang, Sihang Li, Enzhi Zhang, Xiang Wang, Kenji Kawaguchi, Tat-Seng Chua
| Challenge: | Molecular-text modeling is an emerging research field that aims to facilitate molecule-relevant tasks with a textual interface and textual knowledge. |
| Approach: | They propose a new method for reaction-text modeling that uses three types of input contexts to incrementally pretrain LMs. |
| Outcome: | The proposed method improves experimental procedure prediction and molecule captioning and offers competitive results in retrosynthesis. |
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods to control document controllable summarization lack abundant labeled data. |
| Approach: | They propose a question-driven, unsupervised pretraining objective to improve controllability in document controllable summarization tasks. |
| Outcome: | The proposed method outperforms pre-finetuning approaches on QMSum and SQuALITY. |
DocSplit: Simple Contrastive Pretraining for Large Document Embeddings (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing model pretraining methods only consider local information, resulting in low-quality embeddings for large documents. |
| Approach: | They propose a new method which forces models to consider the entire global context of a large document. |
| Outcome: | The proposed method outperforms existing models on document classification, few shot learning, and retrieval tasks. |